Skip to content

Document PerfIO S3 client configuration - #15943

Open
gerashegalov wants to merge 2 commits into
NVIDIA:mainfrom
gerashegalov:fix/15919-s3-timeouts
Open

gerashegalov wants to merge 2 commits into
NVIDIA:mainfrom
gerashegalov:fix/15919-s3-timeouts

Conversation

@gerashegalov

@gerashegalov gerashegalov commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Related to #15919.

Description

The PerfIO S3 reader supports selecting Netty or CRT with the existing
spark.rapids.perfio.s3.httpClient setting, but that setting was omitted from the generated
advanced configuration documentation. The Iceberg S3 async tuning settings are specific to
PerfIO and should not be declared as core RapidsConf entries.

This change:

  • documents the existing PerfIO S3 HTTP client backend selector, including its supported values
    and default;
  • documents the Iceberg S3 response-progress timeout and the optional total-call hard cap. The
    total-call limit is disabled by default because it spans the complete streaming body and can
    abort a healthy but slow transfer; and
  • removes the PerfIO-only Iceberg S3 declarations from the core configuration registry while
    retaining their generated user documentation.

The response-progress timeout is the default protection against stalled streaming range reads.
Operators who require an absolute wall-clock bound can opt in with
spark.rapids.iceberg.s3.async.api-call-timeout-ms.

Validation:

  • ./build/buildall --clean --profile=350 --module=dist --parallel=64 --option='-DskipTests'
  • git diff --check

Performance testing is not required because this change only updates configuration metadata and
documentation; it does not alter request execution.

AI assistance: the implementation and validation were performed with Codex assistance and
reviewed by the author before publication.

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Document the existing PerfIO S3 HTTP client backend selector and the new request timeout settings. Remove PerfIO-only Iceberg S3 declarations from the core configuration registry while retaining their generated user documentation.

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
@gerashegalov gerashegalov added the documentation Improvements or additions to documentation label Sep 9, 2026
@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge, with no concrete correctness, security, or repository-rule violations identified.

Summary

  • Documents NETTY and CRT backend selection with NETTY as the default.
  • Adds and clarifies Iceberg S3 async timeout documentation.
  • Removes duplicate PerfIO-only Iceberg S3 declarations from the core configuration registry.

Reviews (2) · Last reviewed commit: "Document optional PerfIO S3 total timeou..."

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants